Referee Recommendations

This paper quantitatively analyzes referee recommendations at eight prominent economics and finance journals, and the SFS (Society for Financial Studies) Cavalcade Conference, where a known algorithm matched referees to submissions. The behavior of referees was similar in all venues. The referee-specific component in the disposition recommendation was about twice as important as the common component. Referees differed both in their scales (some referees were intrinsically more generous than others) and in their opinions of what a good paper was (they often disagreed about the relative ordering of papers). JEL Code: A14 The editorial process determines not only the evolution of economics and finance but also the incentives and professional fates of academic economists. Yet, its participants do not have much objective knowledge about the process. Authors write only a few papers per year and typically receive only a few referee reports per submission. They usually do not learn which other papers were rejected. They rarely find out why an editor chose a particular referee, much less who the referee was. In turn, they referee only a few papers themselves every year and rarely receive feedback about how their views lined up with those of other referees. The heterogeneity among referee evaluations is further exacerbated by the fact that the journals themselves have also not explicitly stated their objectives and criteria other than in broad and uncontroversial terms. For example, some referees hold the view that only the submitted paper should influence editorial decisions and that fairness to authors is a main goal. Others hold the view that journals should select submissions to maximize their impact, allowing such factors as the identities or institutions of the authors to play a role. However, most economists would agree that it should ideally be submissionand author-associated factors in the broadest sense and not referee-associated factors that should determine publication.1 My paper studies the extent to which referee recommendations reflect a shared consensus versus the extent to which they reflect referee-specific perspectives. If recommendations are relatively more idiosyncratic, then the evolution of knowledge is likely to be more path-dependent (and the careers of economists more random) than if recommendations reflect a general consensus. There could be many reasons why referees share perspectives. They could agree not only with respect to the characteristics of the submissions (such as its novelty, interestingness, accuracy, rigor, and polish, as pointed out by Ellison 2002a), but also with respect to other non-submission-related characteristics (such as the identity of the authors). I shall refer to these aspects as the “reliable qualities” of submissions (not to be mistaken for the 1It is possible that the editorial process is a second-best solution to a moral-hazard problem: editors may have to indulge referee-idiosyncratic opinions in order to incentivize volunteer referees to participate in the editorial process. 1 true scientific quality). Referees could also disagree for other reasons. There could be heterogeneity in their weightings of these characteristics, or there could be referee-specific factors such as noise, skills, time investments, moods, beliefs, ideologies, personal likes, age or cohort, professional networks, vanity, suppression of contrary evidence, or turf motives. Of course, if referees agreed about these characteristics and placed similar weights on them (e.g., if all referees liked pro-free-market papers), these same characteristics would become repeatable commonalities among referees. This would lead my paper to classify these components as “reliable”—again highlighting the difference between reliability (which I can measure) and submission quality (which I cannot measure). It is by this definition that the influence of referee characteristics that are not common (reliable) across referees become idiosyncratic (unreliable). The draw of the referee matters less when the reliable component of referee recommendations plays a more important role than the subjective component.2 An immediate concern in any study that seeks to determine the reliability of referees’ recommendations is that editors do not choose referees randomly. This makes it difficult to determine whether any observed consensus reflects a reliable component of the referees’ views about the submissions or whether it reflects merely the editorial referee selection decision. Therefore, my study examines referee behavior not just in the standard refereeing context (for eight journals: Econometrica [ECMTA], the International Economic Review [IER], the Journal of the European Economic Association [JEEA], the Journal of Economic Theory [JET], the Quarterly Journal of Economics [QJE], the Rand Journal [RAND], the Journal of Finance [JF], and the Review of Financial Studies [RFS]), but also in a conference venue with an unusual referee selection: In the 2012 Society for Financial Studies (SFS) Cavalcade conference, a known computer algorithm matched referees to submissions based only on shared expertise. 2In my paper, I sometimes refer to the idiosyncratic referee-specific aspects as the “subjective evaluation” of the submission. This is not meant to imply that the common aspects do not contain subjective but widely shared views, or that the subjective evaluation cannot be based on objective criteria that only one referee considered. 2 Studying the two venues represents different tradeoffs. On the one hand, human journal editors can presumably match papers better to referee expertise, and journal referees spend more time on journal submissions than on conference submissions. On the other hand, editors may select the number and identities of referees based on their own prior assessments of submission quality or even a desire to influence the referees’ recommendation and/or the agreement among multiple referees. My paper will show that referee behavior is very similar in both types of venues. Without the journal data, the conference data could be viewed as too different from the journal settings. Without the conference data, the journal data could be viewed as the result of deliberate editorial selection. Together, Occam’s razor suggests that my paper documents behavior that is typical of economics and finance referees, and not an artifact of referee selection.